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Abstract: In 1999 Northumbria University published a strategy document entitled “Towards the web-enabled University”. 
This prefaced an assessment of need and of available platforms for developing online teaching and learning which, in 
turn, led in 2001 to the roll out and institution-wide adoption of the Blackboard Virtual Learning Environment (VLE) now 
referred to as our eLearning Platform or eLP. Within a very few years we had over 90% take-up by academic staff and 
the eLP had become integral to the learning of virtually all our students. 

What has always been relatively easy to measure has been the number of users, frequency of use, number of courses, 
levels of technological infrastructure, etc. However, with the publication of the Higher Education Funding Council for 
England (HEFCE) e-learning strategy in 2005 it became apparent that such quantitative data was not particularly helpful 
in measuring how the university matched onto the 10-year aspirations of that document and its measures of success. 

Consequently an on-going exploration was embarked upon to try to measure where we were and what we should 
prioritise in order to embed e-learning, as envisaged within the HEFCE strategy. This involved a number of key 
approaches: 

■ The measures were broken down into manageable sizes, creating sixteen measures in all with descriptors for “full 
achievement” through to “no progress to date” with suggested sources of information which would support the 
description. A series of interviews with key staff were set up in which they were asked to rank where they felt the 
university stood against each measure and what evidence would support their views. 

■ An academic staff survey was developed on-line which invited staff to explore a number of statements based around 
the HEFCE criteria and express degrees of agreement. This was followed up by a range of face-to-face interviews. 

■ An online student survey was developed and students were asked to express degrees of agreement with these. 
Student responses were followed up with an independent student focus group exploring issues in greater depth. 

The outcomes of the three approaches were then combined and an interim report prepared which identified strengths 
and areas for further development. Some of the latter are already being addressed. 

Subsequently, the university joined phase 2 of a national benchmarking e-learning in Higher Education exercise, running 
from May to December 2007, supported by the Higher Education Academy (HEA) and the Joint Information Systems 
Committee (JISC). During this exercise we engaged in a deeper exploration against a wider set of criteria, based upon 
the “Pick & Mix” (Bacsich, 2007) methodology. Pick&Mix comprises 20 core criteria and the option of a number of 
supplementary criteria. Through these approaches we will be able to set a baseline for where we currently are and it will 
allow us to revisit criteria later to measure our progress in those areas we identify for development. 

This paper shares methodologies used, identifies key outcomes and reflects upon those outcomes from both an 
institutional and sectoral perspective. 
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1. Background 

Northumbria University is a large metropolitan university based in Newcastle upon Tyne in the north-east of 
England. It currently has over 1100 academic staff (full- and part-time) and in excess of 32 000 registered 
students. 

In 1999 the university’s senior management group developed a paper, “Towards the web enabled 
university”, which outlined a strategic plan for the development of what is now known as e-learning across 
the university. A major strand in this strategy was the exploration and introduction of a virtual leaning 
environment (VLE) and to that end, a number of potential platforms was explored. By early 2001 the decision 
was made to run a proof of concept exercise with the Blackboard VLE. This proved very successful and 
consequently the VLE was rolled out across the university for academic year 2001-2. Later, in 2003, we 
introduced the fully integrated managed learning environment (MLE). By 2005 we had over 90% take-up by 
academic staff with 32000+ students attached to live sites (Bell & Bell, 2005). Later we introduced the 
integrated content management system and have subsequently undergone further upgrades. 
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The university recognises that e-learning is not simply making use of a VLE but covers a broader spectrum 
of information and communication technologies (ICT). To that end, for the purposes of the exercise 
described here, we have adopted the HEFCE (2005) definition, “The Government e-learning strategy defines 
e-learning as any learning that uses ICT.” (HEFCE, 2005) and, more succinctly from JISC, “Learning 
facilitated and supported through the use of information and communications technology.” (JISC, 2005). 

2. Rationale 

Why did we want to measure our progress in e-learning? After six years during which teaching and learning 
had made integral use of both the VLE and other ICTs it was timely to see where we were and how we 
compared with the sector. A key driver for this came from senior management who wanted to assess how 
well we stood in comparison to three key papers, both external and internal. The first of these is the HEFCE 
e-learning strategy, (HEFCE, 2005) which gave a blueprint of where the successful implementation of e- 
learning would take the UK higher education sector over the succeeding 10 years. The second external 
driver is the government’s Department for Education and Skills (DfES) document, “Harnessing Technology: 
Transforming Learning and Children’s Services” (DfES, 2005) which looks at how e-learning should 
contribute to all education, including school, further education, higher education and life-long learning, setting 
targets for each sector. A third major driver was Northumbria’s own Learning and Teaching strategies (2003- 
2006 and 2007 - 2010) in which e-learning is embedded. 

Finally, in an increasingly competitive market, the university is keenly aware of the need to remain at the 
forefront. To that end we need to be able to measure ourselves against the competition. 

3. What did we want to measure? 

The problem facing Northumbria, in common with other educational organisations, is that many of the things 
which are easy to measure e.g. numbers of users, of courses, page hits etc. only provide quantitative data. 
This kind of data provides information on quantity and extent of use but not on quality and depth of that use. 
Excellence in provision of infrastructure can be gathered from a range of sources, for example, 
Northumbria’s IT provision has led to it being named “the UK's most IT-enabled Organisation” at the 
Computing Awards for Excellence, 2006. Whilst this recognises the excellent work undertaken to ensure 
technical infrastructures are firmly in place, it was important that we also reflected on how effectively that 
technology was impacting upon the learner’s experience and upon teaching and learning across the 
institution, and those qualitative things are what is hard to measure. 

The HEFCE (2005) measures provided a sound framework around which to build our exploration. These 
state: 

“We consider the Higher Education (HE) sector to have embedded e-learning where: 

1. ICT is commonly accepted into all aspects of the student experience of higher education, with 
innovation for enhancement and flexible learning, connecting areas of HE with other aspects 
of life and work. 

2. Due to more coherence and collaboration, technical issues have been addressed to give 
better value for money. 

3. Students are able to access information, tutor support, expertise and guidance, and 
communicate with each other effectively wherever they are. They are able to check and record 
their achievement in a form designed for multiple uses to enable personal and professional 
development. 

4. Tutors have tools for course design to enable better communication between them and their 
students, giving feedback and targeted support. Individual teachers have access to 
information about the materials available, and support for continuous improvement of them. 

5. Subject communities are able to share materials in ways that enhance their ability to produce 
customised high guality courses. They are supported to work collaboratively in designing 
materials, which are effectively guality assured and widely disseminated. They have access to 
research information to inform curriculum development and research-based teaching. 

6. Institutions are able to build appropriate infrastructure and resources support for integrating 
registration and learning functions. They have links with regional networks of institutions to 
support progression and community involvement. 

1. Lifelong learning networks support connectivity between institutions to provide seamless 
access for students and staff. 
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8. Staff are supported at all stages to develop appropriate skills in e-learning, and these skills are 
recognised in their roles and responsibilities and in reward structures. They have access to 
accreditation for their level of skills and professional practice in linking learning technology 
with teaching. ” 

HEFCE strategy for e-learning, March 2005, pg.9 

4. How - the multi-dimensional approach 

At the end of the 2005-2006 academic year we set in train a number of actions in order to try to get to the 
rich data which would allow us to measure our progress against the HEFCE strategy’s aspirations. In order 
to get as full a picture as possible it was essential to involve as many stakeholders as possible. 

Essentially we wanted to know: 

■ What management thought was happening; 

■ What those at the chalk-face experienced; 

■ What those on the receiving end felt, and 

■ What additional evidence supported these perspectives 

4.1 Management 

The HEFCE document was broken into 16 strands (Bell, 2006, pp 19-21) with statements which would 
indicate the degree of progress towards achievement of that element. In addition suggested areas to look for 
evidence which might indicate progress were identified. Respondents were asked to rate progress towards 
the strand using a 3 - 0 scale with 3 being “fully achieved” and 0 being “no progress yet”. 

The second stage was to identify the appropriate people to interview to gain their views on progress towards 
the targets. Accepting that there was no single individual in a position to address all 16 aspects a range of 
key people from across the university were interviewed. Amongst other aspects, these included senior 
colleagues from Learning and Teaching Support, IT Services, Library and Learning Services, Academic 
Registry, Human Resources and the University Learning and Teaching Committee. Evidence in support of 
the responses, including reviewing policies, was then collected. 

4.2 Students 

The HEFCE strategy is very broad in its range and not all sections will have a direct impact upon students. 
Consequently a questionnaire was developed which would address those aspects of the strategy which 
students would be in a position to respond to (see Bell, 2006 p. 23). The survey was carefully constructed to 
gather information around nine aspects of e-learning. For each aspect two conflicting statements were 
developed (this was to ensure validity). These were arranged on the questionnaire in such a way that 
conflicting statements were not adjacent to one another. The survey was then put online and also made 
available in hard copy and students who responded offered entry to a prize draw in order to encourage 
participation. Available responses were “totally agree”, “partially agree”, “partially disagree” and “totally 
disagree” (there was no option to sit on the fence). The survey was made available from April 10 th to June 2 nd 
2006 and elicited responses from 1700 students although some students did not answer all questions. An 
independent qualitative data collection exercise took place during Semester 1 2006-7. Northumbria’s 
Students’ Union was commissioned to undertake this and set up focus groups to gather data. 

The student survey covered the following areas: 

■ How integrally ICT was incorporated in their everyday experience - levels of perceived computer 
use; 

■ Their perception of access to an appropriate range of software; 

■ How easy it was for them to access study materials and support off-campus; 

■ Whether they were able to use ICT effectively for their personal development plans (PDPs); 

■ How effectively the technology enabled them to communicate with their peers (initially there 
were also statements about the levels of contact, feedback and support from tutors but these 
were withdrawn in the light of sensitivity about the industrial dispute as perceived as being 
potentially critical of academic staff); 
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■ How well they felt catered for in terms of open-access ICT facilities; 

■ How easy it was for them to gain access to specialist/course critical software; 

■ How effectively they felt the ICT support infrastructure provided them with help and assistance; 

■ How effectively they felt ICT support worked when they were working off-campus; 


4.3 Staff 

A survey of staff perceptions was begun in June 2006 and was available online until the end of October 
2006. This survey had 15 statements which closely reflected the HEFCE strategy as well as the 16 strands 
of the management audit (see Bell, 2006 p. 24). Respondents select responses from “totally agree”, “partially 
agree”, “partially disagree” and “totally disagree”, again with no option to sit on the fence. Once the survey 
closed a qualitative data collection exercise was undertaken through individual interviews to add further rich 
data to the outcomes. 

4.4 Other sources 

In addition to the surveys and questionnaires described earlier, a broad range of other sources were 
accessed to help provide a broader and deeper picture. These included: 

■ Learning & Teaching Strategy 2003-2006 

■ IT Strategy 

■ Library and Learning Services Strategy 

■ Strategy for research and consultancy 

■ Schools’ Academic Development Plans 

■ VLE statistics, 

■ Staff development statistics 

■ Help-line information 

■ Access statistics 

■ Distance learning courses survey 

■ E-learning enhancement group minutes 

■ Course information 

■ Job vacancies web-site 

■ etc. 

Many of these sources needed to be revisited in the next phase of our exploration, as part of the e-learning 
benchmarking exercise of which we are currently approaching completion. It also enabled us to do some 
drilling down into specific areas of the university’s provision as detailed in section 5.3. 

5. The second phase 

At the same time that we were exploring how we stood against the HEFCE measures, a national programme 
for benchmarking e-learning in Higher Education was under way. In 2006 a pilot exercise took place followed 
by phase 1 which ended in early 2007. Northumbria took the decision that the logical next step for us was to 
become involved in phase 2 of this national project. That phase ran from May 2007 to January 2008. At the 
time of writing we are in the final stages of the exercise. 

5.1 Background to the HEA/JISC benchmarking exercise 

When the HEFCE e-Learning Strategy was published in March 2005, it was a result of a wide-ranging 
consultation with the UK Higher Education community. One of the emerging issues from the feedback 
generated by that consultation was that it would be helpful to institutions to have some kind of benchmarking 
exercise to help them to judge how effectively e-learning had been embedded. 
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Consequently the Higher Education Academy (HEA) put forward a proposal to HEFCE to take the lead in an 
e-learning benchmarking exercise in partnership with a team from the Joint Information Systems Committee 
(JISC). The HEA and JISC were given the go-ahead to embark on the exercise in mid-2005. By the end of 
that year consultants had been appointed and a number of institutions identified to be involved in the initial 
stages of the exercise. 

In December 2006 the call went out for expressions of interest to be involved in phase 2 of the exercise and 
it was at that point that we decided it was both timely and appropriate to become involved. 

5.2 Benchmarking methodology 

The HEA offered a range of benchmarking methodologies (HEA, 2006) and after careful consideration and 
consultation Northumbria chose the Pick&Mix (Bacsich, 2006a) approach. This approach provides a range of 
criteria against which to measure the institution. The methodology comprises 20 core criteria with level 
descriptors from 1 - 5 for each with an additional level 6 to denote excellence. It also offers a range of 
supplementary, optional criteria and it is expected that institutions will choose a further five or six criteria from 
these. However, there is also the opportunity to develop local criteria or to use criteria from other 
benchmarking methodologies (HEA, 2006) or from within the European Association of Distance Teaching 
Universities’ (EADTU) E-xcellence project (EADTU, 2006). 

5.3 Northumbria’s approach 

Once our expression of interest (Eol) had been approved (Bell & Farrier, 2007) we set in train a number of 
actions. The submission of our Eol had the approval and support of an internal committee chaired by the 
Deputy Vice Chancellor (DVC) with responsibility for teaching and learning. It included a management 
structure for our benchmarking exercise. This comprised an advisory panel chaired by the DVC and with 
representation from all key university services (at director level) as well as representatives from schools (at 
associate dean level), chairs of e-learning groups and from the Students’ Union. The purpose of this panel 
was to support and guide the exercise and to provide access to sources of evidence. Alongside this was an 
operational group which included academic staff, a researcher and project assistant. This was the group 
which actually undertook the work. Resources for the exercise were principally provided internally with some 
input from HEA external consultants. 

In terms of selecting supplementary criteria, a consultation exercise was undertaken and feedback from that 
enabled us to identify those six additional criteria which are of particular relevance to our institution. These 
relate to uniformity, ubiquity, widening participation, personalisation, student satisfaction and e-portfolios. 

To make the whole process manageable we undertook an exercise in order to “chunk” the criteria into 
achievable work packages. Two issues had a major bearing upon how the chunks were organised. Firstly we 
took into account the sources of evidence which would need to be explored in order to address the criteria. 
For example, some criteria require interviews to take place with members of senior management. These 
were chunked together to avoid the need for multiple interviews with the same people. Other criteria clearly 
lend themselves to desk research, and so on. This provided us with seven chunks which were then pulled 
together into six work packages: 


Work package 

Criteria (numbers) 

WP1 

04 usability and 05 accessibility 

WP2 

02 VLE stage, 07 decisions (projects), 19 decisions (programmes) 20 Quality enhancement 

WP3 

08 pedagogy, 14 evaluation, 17 quality assurance, 18 staff recognition and reward, 95 e- 
portfolios 

WP4 

06 e-learning strategy, 15, organisation, 72 personalisation, 94 student satisfaction 

WP5 

03 tools, 09 learning material, 16 technical support to staff, 51&52 uniformity and ubiquity, 70 
widening participation 

WP6 

01 adoption, 10 training, 11 academic workload, 12 costs, 13 planning (annual) 


NB - criteria numbers relate to those in Pick&Mix - version 2.0 (Bacsich, 2006) 


Work packages were then matched on to the particular expertise and strengths of operational group 
members. However, we recognised the need to remain flexible and it later proved desirable to revise the 
original work packages in the light of changes in staffing, pressures of workload and other responsibilities. It 
was also essential that each work package “owner” was supported by other members of the operational 
group and that close attention was paid to where evidence could be used to support outcomes of other work 
packages. 
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After attending an initial kick off meeting hosted by the HEA, Northumbria was joined by nine other UK 
institutions using the same methodology and this cohort was supported by external consultants employed by 
the HEA specifically to support the benchmarking process. In addition to organising a series of cohort 
meetings, part of the consultants’ roles has been to visit participating institutions several times during the 
exercise to support the process and to assist in the final scoring activity. Consultants also worked on the 
identification of sector-wide areas for development. 

Given the tightness of the timescale (May 2007 to January 2008) the work was very intensive and one of the 
key early activities was to undertake a preliminary scoring activity which then provided us with an early 
indication of where in depth research would need to be focussed. If, for example, we allocated a score of 3 to 
criterion 95, e-portfolios, then how well did the evidence support this and might the reality actually be 
different? 

A second priority concerned identifying areas where it was appropriate to undertake a “slicing” activity where 
an individual element of provision, be it a school, an approach to delivery (e.g. collaborative working with 
partner institutions) or even a service (e.g. library services). At this point we identified two schools with the 
potential for slicing and who were keen to undertake this, and a delivery area which concerned partnership 
franchises and collaborative ventures. 

Interim outcomes from our initial exercise were reported in late 2006 (Bell, 2006) and final outcomes of the 
HEA benchmarking exercise are being reported in January 2008. Since the benchmarking exercise should 
provide us with a base-line assessment of our position, it will enable us to review future progress and 
development on a regular basis and it is our intention that the exercise be repeated, probably on a two- or 
three-yearly basis. 

5.4 Data gathering 

It was essential before we embarked upon the benchmarking exercise to engage in an interpretation of the 
criteria in the Northumbria context and to identify likely sources of data/information. This exercise was 
valuable since criteria were, in places, rather too generic. 

The data gathering exercise began in earnest in June 2007 but hit a hiatus in late July and August due to the 
summer break. Much management, staff and student feedback was already available based upon 
questionnaires, interviews and focus group reports and this meant that the main focus for research was the 
documentary evidence - policies, minutes, papers, strategies - to support the investigation with interviews 
and focus groups being available if needed. In addition a presentation was given at the annual Northumbria 
Learning and Teaching Conference where participants were involved in engaging with the criteria. 

An evidence room was set up for the documentation to be stored in appropriate criterion boxes and this 
resource continues to be built upon continuously as new policies are released and initiatives engaged in. 
Additionally, the bulk of “discovered” evidence is also available electronically, although a proportion is only 
available internally to the university. 

By mid-November criterion sheets were completed for all criteria, although the team recognised that this was 
an arbitrary cut-off point, in order that they could be collated, edited and distributed in time for the final 
scoring meeting. For each criterion there were four main sections - 

■ Criterion interpretation and key questions; 

■ Sources of evidence (with links where appropriate); 

■ A commentary on case studies 

■ Two exemplars, one of typical practice and one of effective, innovative or noteworthy practice. 

The final scoring meeting was held in late November 2007, chaired by a senior member of university staff 
with both advisory panel and operational team members contributing, along with an external consultant. This 
then allowed the initial preparation of reports for a range of audiences - internal, public, the HEA and the 
consultants. The drafting and redrafting process is currently nearing completion. 
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6. Initial findings: 

The first stage of the exploration revealed a number of issues and indicators for future development and 
review. When allied to the outcomes of the benchmarking exercise, they indicate that Northumbria has a 
number of strengths in e-learning. Although the initial work packages offered coherence in terms of 
managing the exercise, when reviewing outcomes and areas the criteria cover we found it helpful to group 
them into five categories. Whilst there are overlaps in many cases, the categories offer a potentially coherent 
way forward. 

The five categories are: 

1. Organisational and technical infrastructures and strategic planning; 

2. Quality assurance, enhancement and evaluation; 

3. Staff experience, support and development; 

4. The learning journey; 

5. Student experience and support. 

The diagram below illustrates how the categories interact with the Pick&Mix criteria. 


Criteria by category 



Our findings indicate that Northumbria has a great deal to celebrate. The Computing Award for Excellence, 
2006, reflects the breadth and strength of the underlying technical structures put in place by the university, 
including the VLE, campus-wide wireless access, thin client architecture, open access labs and excellent 
electronic library resources. In addition both staff and student surveys indicate a high level of satisfaction 
showing that ICTs form part of the everyday teaching and learning experience. The virtual universal take-up 
of the VLE and its on-going development indicate, again, the ubiquity of ICT use across the institution. 
Summarising some of the interim and final findings, the following areas are highlighted: 

■ Organisationally and technically there are many strengths. The range of ICT tools and facilities 
available to learners and teachers is excellent and the university has demonstrated expertise in 
the implementation of major elearning projects. At the same time sound, interconnected 
strategies have been developed incorporating elearning across a broad range of these. An 
excellent set of structures is in place to support staff training and development. Decision making 
for elearning developments is embedded within the work of schools, departments and subject 
divisions. 

■ Areas for further development include the need to incorporate elearning expertise into staff roles 
and into the staff recognition process. Furthermore, emphasis upon the different emerging 
pedagogies relating to online learning could form a key strand in future staff development to 
ensure effective development of new delivery approaches for programmes and modules. In 
common with many other Higher Education Institutions (HEIs), costing models and workload 
models which take into account the extra requirements for elearning have yet to be developed. 
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Further, we do not currently have a single locus for the coordination of developments within the 
elearning field. 

■ With respect to quality assurance, enhancement and evaluation, robust systems are in place to 
ensure that online teaching and learning materials adhere to appropriate standards particularly 
of accessibility and usability. Evaluation of provision is well regulated and effective. Approaches 
to quality enhancement are under constant review and needs responded to. 

■ In common with many similar institutions, ensuring current systems and approaches keep pace 
with elearning can be problematic. There is a need to ensure that staff development reflects this 
shifting arena. The burgeoning growth of online materials and delivery can present quality 
assurance difficulties and this is an area for further exploration and development. Indeed, many 
systems which are effective for conventional teaching may need to be reviewed to take account 
of the changes elearning brings. 

■ Staff at Northumbria have wholeheartedly adopted the use of our VLE and a range of rewards 
are available which recognise expertise in the area. There is good central support available to 
staff to assist in the development of elearning and in some areas of the university elearning 
development is recognised as being time-consuming and allowances are made for this. 

■ We recognise now the need to move on from baseline adoption where the VLE is used by a 

proportion of staff as a document repository and encourage the use of the technology for 

collaboration, communication and interaction within elearning. The changing and emerging 
pedagogies associated with this will require embedding. Greater access to support will become 
essential to engage all staff in using elearning at a deeper level. 

■ Our students’ learning journey is greatly enhanced through the use of online learning. Most 
provision matches well the Neilson (2003) usability criteria and access to elearning is both 
ubiquitous and uniform across all provision, both campus-based and beyond in partner 
institutions. 

■ There are, though, areas for development. In common with many other institutions engaged in 

the benchmarking exercise, accessibility is an issue. Efforts made to ensure software is 

accessible are to be celebrated but the same level of vigilance could be applied to all materials 
offered to learners within the VLE and elsewhere. There is also a need to ensure that, with 
expanding provision, access and uniformity of provision is maintained. 

■ Our students are very positive about their elearning experience. The student survey and other 
student feedback indicates that students are very happy with both the quality and quantity of 
facilities and see ICT as integral to their studies, with 89% expressing satisfaction with the 
former and 96% using ICT in their daily studies. They are also happy with the levels of support 
available. 

■ Two areas in particular suggest the need for further development. The full potential for e- 
portfolios for personal development planning is an area currently being explored. Given recent 
EU and national aspirations, on-going development is desirable. Secondly, the disparity found by 
students between different module sites within the VLE is of concern and consideration needs to 
be given as to how equality of provision can be maintained. This relates to consistency of 
approach and of use by academic staff. 

The Benchmarking Exercise has revealed that Northumbria’s approach to elearning (and indeed to all 
learning and teaching) has a great deal to commend itself upon. What is also evident is that there are 
opportunities for further development which would contribute towards the delivery of an outstanding 
experience for all students and staff. 

7. Initial responses 

Following the preparation of the interim report presented to an e-learning strategic management group in late 
2006 some actions have already been begun. 

Reflecting on the patchy take-up electronic personal development plans (PDPs) an immediate response was 
to develop a strategy to encourage the development of these and to provide appropriate staff development. 
In terms of quality assurance of taught courses, we are actively reviewing the processes we have in place for 
validation of programmes, with an initial focus upon distance and blended learning. A key area for action 
concerned staff development and currently a review of our approaches to this has begun with the 
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expectation that this will lead to greater coherence in this provision. Related to this, a review of how we 
recognise and reward staff skills and expertise in e-learning is being planned. In a further development, 
students are now able to access their record of achievement online. 

The preparation of the new L&T strategy saw e-learning embedded within it and will be accompanied by the 
development of a number of implementation plans with one specifically for e-learning. In addition, individual 
schools’ academic development plans will need to indicate how e-learning will impact upon their future work. 

These are practical outcomes from the first phase of our measurement exercise. The second phase has 
highlighted a number of additional areas of strength and areas for development (see above). It is the’s 
intention to prioritise these and to that end a process has begun which will involve a broad range of 
stakeholders in exploring and identifying where priorities are and where responsibility for their further 
development lies. Subsequently implementation plans will be put in place and actions reviewed on a periodic 
basis, leading to a repeat of the process in three years’ time. 

8. Lessons 

Engaging with the benchmarking process has a number of major benefits for the university. It has helped us 
to celebrate strengths and identify areas for further development. This is particularly relevant as elearning 
imposes different needs and constraints upon the planning, preparation, delivery and maintenance of 
learning and teaching situations. 

The slicing aspect of benchmarking has provided the opportunity to explore local practice within the overall 
university context. One of the difficulties which arise in institutions of this size is the tendency for a silo effect 
to take place. The benchmarking exercise has allowed us to explore and celebrate effective practice across 
schools and should enable and encourage institution-wide dissemination. Slicing is still on-going and 
expected to report in early 2008. 

In the same way, the identification and use of exemplars of both typical and innovative or noteworthy 
practice provides a picture of what is standard practice and what is achievable. It has helped to highlight 
areas for development and the art of the possible. 

One of the difficulties we encountered when engaging for the first time with the HEFCE measures of success 
was that they were not broken down into manageable statements and needed time spent in order to make 
sense of them in a way which allowed comparison. Furthermore, it proved necessary to create a scoring 
system to apply to the measures. The advantage with the Pick&Mix methodology was that it had a recent 
history of development, had already been used in two earlier phases of the exercise and had, consequently, 
gone through an iterative process leading to a better focus and refinement. This meant that the methodology 
was quite straightforward to use. However, against that, because the methodology had been refined, it could 
become something of a straightjacket. As detailed in section 6.1, a number of criteria need to be reviewed. 
Further criteria which focus upon the student experience need to be incorporated and the rigidity of the core 
criteria would benefit from relaxation. Interpretation of criteria can also be problematic and a 
recommendation to the consultants would be to revisit core criteria and ensure their meaning is clearer. 

An interesting consequence of the exercise has been the opportunity to engage with the broader HE 
community. Within our specific cohort was a range of HE providers, from small focussed institutions to large 
institutions like ours. This meant that a variety of perspectives have been available. Interestingly, we have 
discovered a great deal of common ground in terms of areas for development and this has created the 
potential for collaborative activities. 

Reflecting on process, a number of lessons have been learned and future benchmarking activities will benefit 
from these. 

1. In future it would be advisable to create a clear division between project management/coordination and 
the collection of data. The complexity of the process and the changing availability of staff meant that 
there was limited slack and the project management ended up carrying a larger burden than was 
comfortable in terms of hands on data collection. 

2. We discovered that a more effective approach than simply carving up criteria was to approach from two 
directions at the same time - criteria identifying and interrogating data and also data “tagging” criteria. 
This helped reduce duplication and repetition and speeded up the process. 
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3. Reliance on an external methodology was, in places, inhibiting. In future we would take only those 
elements which were appropriate to our context and cover other elements based upon our own criteria. 
We need to be able to find out what we need to know rather than cater to the needs of an external driver. 

4. This exercise was partly to enable the HEA and JISC to gain a broad-brush impression of the state of 
elearning across the sector. Any future benchmarking activity can focus upon the specific needs of 
Northumbria, relating closely to its policies and strategies and reflecting upon these in the light of national 
strategies. 

It is also important to recognise the risks inherent in undertaking an exercise of this nature. Firstly, if 
undertaken in a totally impartial way, it can reveal uncomfortable disconnects within overall provision. 
Secondly, although the exercise is specifically about elearning, it is revelatory of the much broader learning 
and teaching practice identifying areas of concern which are not confined to elearning alone. Thirdly, there is 
a danger that, once completed, actions will be undertaken to tackle the “easy” areas for development and 
others may be left in abeyance or ignored. In a large institution like ours there is also a danger of the buck 
being passed and only limited action taken as the issues are “somebody else’s problem”. 

However, the risks of not reviewing provision in such a robust way are also great. In the current highly 
competitive global higher education market, provision which fails to match need will suffer. By undertaking an 
exercise of this nature, institutions can ensure relevance and currency which match learners’ needs. 

9. Conclusion 

Although the measurement exercise has a largely internal focus we are also committed to disseminating 
outcomes more widely and, particularly through the HEA. In places the tools used are very specific to the 
UK, but the process and many of the associated actions are of relevance to all educational institutions who 
are engaged in aspects of e-learning. Interestingly, we have found that many of the criteria and measures we 
have used, with some additions, could be applied to learning as a whole. A broad version of the Pick&Mix 
criteria has been developed incorporating an additional 11 supplementary criteria with the specific intention 
that it can be applied to all learning and teaching, not only that which is prefixed with “e-“. 

For Northumbria University, measuring our progress in e-learning has already helped us to focus upon areas 
for further development as well as areas of strength. The continuing cycle of improvement in which the 
university engages provides benefits for the whole institution, individual areas within it and for the wider HE 
community in the UK. 
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